Interactive design of multimodal user interfaces : reducing technical and visual complexity

In contrast to the pioneers of multimodal interaction, e.g. Richard Bolt in the late seventies, today's researchers can benefit from various existing hardware devices and software toolkits. Although these development tools are avai lable, using them is still a great challenge, particularly in terms of their usability and their appropriateness to the actual design and research process. We present a threepart approach to supporting interaction designers and researchers in designing, developing, and evaluating novel interaction modalities including multi modal interfaces. First, we present a software arc hitecture that enab les the unification of a great variety of very heterogeneous device drivers and special-purpose toolkits in a common interaction library named "Squidy". Second, we introduce a visual design environment that minimizes the threshold for its usage (easeof-use) but scales well with increasing complexity (ceiling) by combining the concepts of semantic zooming with visual dataflow programming. Third, we not only support the interactive design and rapid prototyping of multi modal interfaces but also provide advanced development and debugging techniques to improve technical and conceptual solutions. In addition, we offer a test platform for controlled comparative evaluation studies as well as standard logging and analysis techniques for informing the subsequent design iteration. Squidy therefore supports the entire development lifecyc\e W.A. Konig (r8J) . R. Radle · H. Reiterer Human-Computer Interaction Group, University of Konstanz, Box D-73, Constance, Germany e-mail: [email protected] uri: http://hci .uni-konstanz.de R. Radle e-mail: Roman.Raedle@ uni -konstanz.de H. Reiterer e-mail: [email protected] of multi modal interaction design, in both industry and research.